Systems and methods for creating and selecting models for predicting medical conditions

ABSTRACT

Systems and methods are provided for selecting one or more models for predicting medical conditions. An exemplary method may include receiving data related to a patient and extracting metadata from the received data. The method may further include selecting the one or more models from a library of models based on the extracted metadata and applying the selected one or more models. The method also may include generating a notification when the application of the selected one or more model indicates an intervention is necessary.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application claims the benefit of priority from ProvisionalApplication No. 61/839,528, entitled “Systems and Methods for ManagingPatient Conditions,” filed Jun. 26, 2013, which is incorporated hereinby reference in its entirety. This Application is also related to U.S.Application No. 13/428,763, entitled “Adaptive Analytical HealthAssistant System and Related Methods of Use,” filed Mar. 23, 2012, whichis incorporated herein by reference in its entirety.

TECHNICAL FIELD

Various embodiments of the present disclosure relate generally tocreating and selecting models for predicting medical conditions. Morespecifically, particular embodiments of the present disclosure relate tosystems and methods for first creating a library of models that predictmedical conditions. When new data from a patient is received, models maybe selected from a library of models to predict medical conditionsincluding adverse health events.

BACKGROUND

An important aspect of medical care is disease management. For effectivedisease management, it is not only important to understand the cause ofany medical episodes related to the underlying medical conditions andtaking suitable actions, but it is equally important to be able topredict any future episodes requiring medical intervention beforehand.

With the advent of sensors and health monitoring technologies that areuser friendly, the collection of health and behavior data has increasedsignificantly. However, prevailing methods fail to accurately predictfuture episodes requiring medical intervention.

For example, diabetes is a result of poor control over one’s bloodglucose (BG) levels. Two major forms of diabetes are type 1 and type 2.Type 1 occurs from the body’s failure to produce insulin, and requiresthe person to inject insulin or wear an insulin pump. Type 2 is a resultof body’s resistance to insulin, a condition in which cells do notproperly use insulin, sometimes combined with an absolute insulindeficiency.

One of the major complications of diabetes resulting in acute episodesis called hypoglycemia. An episode refers to when a person’s BG leveldrops below approximately 70 mg/dL. Such an episode results ininadequate supply for glucose to the brain which may cause seizures,unconsciousness, mild dysphoria, and may even result in death if nottreated immediately. Hypoglycemia may be caused by numerous factorsincluding an insulin overdose, inadequate nutrition, lack of nutritionbefore exercise, natural metabolism, abnormal blood glucose values,amount of nutrition, and medication.

Hypoglycemia may be treated by restoring BG levels to normal bymedication, nutrition or glucose tablets. Major factors causinghypoglycemia are overdosing of medications, inadequate nutrition, andlack of nutrition before exercise and due to one’s metabolism. However,data related to such factors is often not available from patients. Whileprevailing techniques use continuous glucose monitoring devices (CGM) toimmediately respond to such episodes. These techniques do not allow foraccurate predictions of future hypoglycemic episodes based onpatient-reported self-monitored blood glucose (SMBG) data.

SUMMARY OF THE DISCLOSURE

Embodiments disclose systems and methods for creating and selectingmodels for predicting medical conditions, specifically medical episodesrequiring interventions.

According to some embodiments, computer-implemented methods aredisclosed for selecting one or more models for predicting medicalconditions. In an exemplary embodiment, the method includes receivingdata related to a patient, extracting metadata from the received data,and selecting the one or more models from a library of models based onthe extracted metadata. The method further includes applying theselected one or more models and generating a notification when theapplication of the selected one or more model indicates an interventionis necessary.

According to some embodiments, systems are disclosed for selecting oneor more models for predicting medical conditions. One system includes amemory having processor-readable instructions stored therein and aprocessor configured to access the memory and execute theprocessor-readable instructions, which when executed by the processorconfigures the processor to perform a method. In an exemplaryembodiment, the method includes receiving data related to a patient,extracting metadata from the received data, and selecting the one ormore models from a library of models based on the extracted metadata.The method further includes applying the selected one or models andgenerating a notification when the application of the selected one ormore model indicates an intervention is necessary.

According to some embodiments, a non-transitory computer readable mediumis disclosed as storing instructions that, when executed by a computer,cause the computer to perform a method, the method includes receivingdata related to a patient, extracting metadata from the received data,and selecting the one or more models from a library of models based onthe extracted metadata. The method further includes applying theselected one or models and generating a notification when theapplication of the selected one or more model indicates an interventionis necessary.

According to some embodiments, the methods may further include receivingdata related to the patient’s blood glucose level, extracting metadataindicating whether the patient’s blood glucose level is continuousglucose monitor data or self-monitored blood glucose data, selecting afirst type of model when the extracted metadata indicates that thepatient’s blood glucose level is continuous glucose monitor data andselecting a second type of model when the extracted metadata indicatesthat the patient’s blood glucose level is self-monitored blood glucosedata. The method may further include selecting the second type of modelwhen the extracted metadata indicates that the patient’s blood glucoselevel is continuous glucose monitor data and self-monitored bloodglucose data, and transmitting the generated notification to one or moreuser devices where

Additional objects and advantages of the disclosed embodiments will beset forth in part in the description that follows, and in part will beapparent from the description, or may be learned by practice of thedisclosed embodiments. The objects and advantages of the disclosedembodiments will be realized and attained by means of the elements andcombinations particularly pointed out in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate various exemplary embodiments andtogether with the description, serve to explain the principles of thedisclosed embodiments.

FIG. 1 is a schematic diagram of a network environment for selecting oneor more models predicting medical conditions, according to an embodimentof the present disclosure.

FIG. 2 is a flow diagram of an exemplary method 200 for creating one ormore models, according to an embodiment of the present disclosure.

FIG. 3 is a flow diagram of a method 300 for selecting one or moremodels predicting medical conditions, according to an embodiment of thepresent disclosure.

FIG. 4 illustrates. an exemplary use case applying the principles of amethod, according to an embodiment of the present disclosure.

FIG. 5 is a block diagram of an exemplary computer system in whichembodiments of the present disclosure may be implemented.

DESCRIPTION OF THE EMBODIMENTS

While the present disclosure is described herein with reference toillustrative embodiments for particular applications, it should beunderstood that embodiments of the present disclosure are not limitedthereto. Other embodiments are possible, and modifications can be madeto the described embodiments within the spirit and scope of theteachings herein, as they may be applied to the above-noted field of thepresent disclosure or to any additional fields in which such embodimentswould be of significant utility.

In the detailed description herein, references to “one embodiment,” “anembodiment,” “an example embodiment,” etc., indicate that the embodimentdescribed may include a particular feature, structure, orcharacteristic, but every embodiment may not necessarily include theparticular feature, structure, or characteristic. Moreover, such phrasesare not necessarily referring to the same embodiment. Further, when aparticular feature, structure, or characteristic is described inconnection with an embodiment, it is submitted that it is within theknowledge of one skilled in the art to effect such feature, structure,or characteristic in connection with other embodiments whether or notexplicitly described.

In view of the challenges associated with the conventional techniquesoutlined above, systems and methods are disclosed herein for creatingand selecting models for predicting medical conditions.

Reference will now be made in detail to the exemplary embodiments of thedisclosure, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers will be usedthroughout the drawings to refer to the same or like parts.

FIG. 1 is a schematic diagram of an exemplary network environment inwhich various systems may select one or more models predicting medicalconditions (including, but not limited to, adverse health events),according to an embodiment of the present disclosure. As shown in FIG. 1, the environment may include a plurality of user or client devices 102that are communicatively coupled to each other as well as one or moreserver systems 106 via an electronic network 100. Electronic network 100may include one or a combination of wired and/or wireless electronicnetworks. Network 100 also may include a local area network, a mediumarea network, or a wide area network, such as the Internet.

In one embodiment, each of user or client devices 102 may be any type ofcomputing device configured to send and receive different types ofcontent and data to and from various computing devices via network 100.Examples of such a computing device include, but are not limited to,mobile health devices, a desktop computer or workstation, a laptopcomputer, a mobile handset, a personal digital assistant (PDA), acellular telephone, a network appliance, a camera, a smart phone, anenhanced general packet radio service (EGPRS) mobile phone, a mediaplayer, a navigation device, a game console, a set-top box, a biometricsensing device with communication capabilities, or any combination ofthese or other types of computing devices having at least one processor,a local memory, a display (e.g., a monitor or touchscreen display), oneor more user input devices, and a network communication interface. Theuser input device(s) may include any type or combination of input/outputdevices, such as a keyboard, touchpad, mouse, touchscreen, camera,and/or microphone.

In one embodiment, each of the user or client devices 102 may beconfigured to execute a web browser, mobile browser, or additionalsoftware applications that allows for input from patients and otherindividuals from the medical field including physicians, nurses,pharmacists, etc. One or more of user or client devices 102 may befurther configured to execute software allowing for monitoring ofpatient related data including the ability to receive user input orutilizing associated sensors to monitor patient conditions. For example,user or client devices 102 may contain an application which allows it toreceive data from a paired and/or integrated blood sugar level monitorand then transmit the data to other entities within environment 100.Alternatively, user or client devices 102 may contain applications whichallow for a patient to input information related to patient behaviorsuch as consumption of nutrition and medication.

Server system 106 in turn may be configured to receive data related toone or more patients including data related to patients’ health ormedical condition, including information related to prescribedmedication regimens, vitals, chronic diseases, nutritional requirements,nutrition consumption, medication consumptions, etc. It should be notedthat while a singular server system 106 is described, method 200,described below with respect to FIG. 2 , may be implemented using aplurality of server systems working in combination, a single serverdevice, or a single system.

Also, as shown in FIG. 1 , server system 106 may include one or moredatabases 108. In an embodiment, databases 108 may be any type of datastore or recording medium that may be used to store any type of data.For example, databases 108 may store data received by or processed byserver system 106 including information related to a patient and or alibrary of models which aid in predicting medical conditions.

Additionally, as shown in the example of FIG. 1 , server system 106 mayinclude processor 110. In an embodiment, processor 110 may be configuredto execute a process for creating and selecting models for predictingmedical conditions. The management process may, for example, continue toreceive patient data, generate models utilizing data related topatients, and then select models for predicting medical conditions.

In an embodiment, processor 110 may be configured to receiveinstructions and data from various sources including user or clientdevices 102 and store the received data within databases 108. In someimplementations, any received data may be stored in the databases 108 inan encrypted form to increase security of the data against unauthorizedaccess and complying with HIPAA privacy regulations. Processor 110 orany additional processors within server system 106 also may beconfigured to provide content to client or user devices 102 for display.For example, processor 110 may transmit displayable content includingmessages or graphic user interfaces soliciting information related topatient behavior. Additionally, the messages may indicate that anintervention is necessary due to a predicted medical condition

A first aspect of the exemplary method 200 entails creating a librarywith models which can predict medical occurrences. Specifically, FIG. 2is a flow diagram of an exemplary method 200 for creating one or moremodels for predicting medical conditions, according to an embodiment ofthe present disclosure

In step 202, method 200 includes receiving data related to one or morepatients. The received data may include receiving data related topatients’ health or medical conditions, including information related toprescribed medication regimens, vitals, chronic diseases, nutritionalrequirements, nutrition consumption, medication consumptions, etc. Forexample, for patients dealing with hypoglycemia, the received data mayinclude information related to a patient’s blood glucose level, amountsof nutrition consumed, time of the day associated with checking bloodglucose levels and consumption of nutrition and times associated with apatient’s medical regimen.

In embodiments, the data may be received based on self-reporting bypatients. Alternatively, devices such as continuous glucose monitoringdevices (CGM) or self-monitoring of blood glucose devices (SMBG) maytransmit this data to server 106 automatically. In embodiments, SMBGdata may have certain advantages over CGM data including more accuracyeven if it is significantly sparser than CGM data. The more accurateSMBG data may be more readily available from devices configured toautomatically upload patient data to servers 106.

In step 204, method 200 may include cleansing the received data. Indetail, raw data is transformed into a form that is easily readablewithout ambiguity for a machine learning algorithm. In embodiments, thecleansing involves determining the presence of anomalies, errors,missing values, and large number ranges etc. in the received data andfixing them. The cleansing may be conducted by an automated processdefined by rules replicating manual data cleansing by a data analyst. Inother embodiments, a data analyst may input instructions for manuallycleansing the received data.

In embodiments, cleansing the received data allows machine learningalgorithms to more readily understand underlying patterns in the data.To minimize the introduction of manual bias based on modifying theoriginal pattern in raw data, missing value instances may be replacedwith indicators such as ‘N/A’ or ‘0’. In embodiments, this may help themachine learning algorithm in understanding that a value should haveexisted in that location and while the value may be ignored, the factthat such a value existed may be considered during its learning process.

In some embodiments, cleansing the data may refer to using only someparts of the received data or transforming the received data. Forexample, for diabetes patients, the only relevant aspects of data forproviding to machine learning algorithms may be the blood glucose levelsand the associated time stamps. In such a scenario, the rest of receiveddata related to the patient may be removed from consideration. In someembodiments, the time stamp may be used to calculate other variable suchas time of the day, date, month, etc. Additionally, duration betweensuccessive values of blood glucose levels may be determined.

In step 206, method 200 includes applying machine learning algorithms tolearn from the cleansed data. In instances, a certain percentage of datapoints may be utilized as a training set for the machine learningalgorithm. For example, out of a total 1000 data points by 100 patients,500 data points may be used for training/learning while the rest of the500 data points may be used for testing.

In embodiments, machine learning algorithms learn all possible patternswithin the patient data points. The machine learning algorithms mayfurther learn about patterns between additional factors as well as therelationship of those patterns with a prediction variable. Theprediction variable is the variable that is to be predicted utilizingthe machine learning algorithms. Every algorithm has parametric valuesthat define a configuration of the algorithm that best predicts theprediction variable. Such configurations are called models.

In step 208, method 200 includes testing on different data. Continuingthe example from before, the 500 data points for testing may be appliedto a model. The machine learning algorithms predict a predictionvariable with a level of confidence. The level of confidence mayindicate how accurate the prediction variable may be. In embodiments, todetermine effectiveness of the models, the models need to be tested withvarious different data sets.

In embodiments, machine learning algorithms, include, but are notlimited to support vector machines, k-nearest neighbor, Bayesianstatistics (e.g., native Bayes), multi-layer perceptron’s, andmultivariate regressions.

In step 210, method 200 includes evaluating models based on the resultsof testing on different data of step 208. The prediction variable valuesthat are removed before testing may be used as a reference. The valuespredicted by the model during testing may then be matched with thereference values to identify cases of data when the model did and didnot predict correctly. The evaluation of whether the model led to anaccurate prediction or not may be expressed in percentages. A model’ssuccess and failure in prediction may be expressed in the form of atable called a confusion matrix. For example, a confusion matrix mayinclude values predicted by using a model and the actual (reference)value. These values may be utilized to determine a precision percentagewhich may also be included in the confusion matrix.

A model or plurality of models may be stored in a library of models (notillustrated) in databases 108. The models in the library of models maythen be utilized by processor 110 to predict a need for medicalintervention when new patient data is received.

As an example of a use case, consistent with embodiments of the presentdisclosure, 1000 blood glucose data points and respective timestamps maybe received or stored (in databases 108) related to a 100 patients. The1000 data points may be then split in half (or any suitable function),where 500 may be used for training the algorithm and the other 500 maybe used for testing the models. If it is known that 250 of 500 datapoints utilized in testing are hypoglycemia events and a model predictsall of the 250 instances accurately, than the model is 100% accurate.

In such a scenario, with an exemplary model with 100% accuracy, when apatient provides 20 new blood glucose data points, application of themodel may accurately predict if the patient may encounter a hypoglycemicevent the next day and if so at what time of the day. If a hypoglycemicevent is predicted for the next day, this information may be used togenerate a notification that medical intervention may be necessary. Forexample, the notification may inform a system (not illustrated)controlling a patient’s insulin pump to regulate an appropriate amountof insulin at the right time before the occurrence of the predictedevent.

Further details regarding assessing performance of machine learningalgorithms and selecting models are provided accordingly to exemplaryembodiments. An important aspect in utilizing an appropriate machinelearning algorithm is the interplay between bias, variance and modelcomplexity.

A first aspect of the process is determining a quantitative response.For example, there may be a target variable Y, a vector of inputs X, anda prediction model f^(X) that has been estimated from a training set T.

The loss function for measuring errors between Y and f^(X) may bedenoted by L(Y, f^(X)), where:

$L(Y,\mspace{6mu} f^{\hat{}}(X)) = ((Y - f^{\hat{}}(x))2\mspace{6mu}\mspace{6mu}\mspace{6mu}\mspace{6mu} squared\mspace{6mu} error\mspace{6mu}$

$L(Y,\mspace{6mu} f^{\hat{}}(X)) = ((Y\, - \, f^{\hat{}}(X))\mspace{6mu}\mspace{6mu} absolute\mspace{6mu} error$

Test error, also referred to as generalization error, may be theprediction error over an independent test sample:

$\left. Err = E\lbrack L(Y,f^{\hat{}}(X)) \middle| T\rbrack \right.$

Where both X and Y are drawn randomly from their joint distribution(population). Here the training set T is fixed, and test error refers tothe error for this specific training set. A related quantity is theexpected prediction error (or expected test error):

$Err = E\lbrack L(Y,f^{\hat{}}(X))\rbrack = E\lbrack ErrT\rbrack.$

Note that this expectation averages over everything that is random,including the randomness in the training set that produced f^.

Training error is the average loss over the training sample:

$eRR = (1/N){\sum{errNi = 1\,\, L(yi,f^{\hat{}}(xi)).}}$

It may be helpful to determine the expected test error of the estimatedmodel f^(^). As the model becomes more and more complex, it may use thetraining data more and may be able to adapt to more complicatedunderlying structures. Hence, there is a decrease in bias but anincrease in variance. Accordingly, there may be provided intermediatemodel complexity that provides minimum expected test error.

Unfortunately training error may not be a good estimate of the testerror. Training error consistently decreases with model complexity,typically dropping to zero when the model complexity is increasedenough. However, model with zero training error may be overfit to thetraining data and will typically generalize poorly.

To continue the hypoglycemia example, in the case of hypoglycemiaprediction there may be a nominal/categorical response. Similarly, for aqualitative or categorical response G taking one of 2 values in a set G,labeled for convenience as k=0,1 (hypoglycemia YES/NO). Probabilities pk(X) = Pr(G = k|X ) may be modeled, where the loss functions are:

$L(G,G^{\hat{}}(X)) = I(G \neq G^{\hat{}}(X))\,\,\,\,\,\,(0 - 1\,\, loss),$

$L(G,p^{\hat{}}(X))\,\,\,\, = \,\,\, - 2{\sum{2i = 1\,\,\,\, I(G = k)\,\,\log\,\, p}}^{\hat{}}k\,(X)$

$\,\,\,\,\,\,\,\,\, = - 2\,\,\log\, p^{\hat{}}G(X)\,\,\,\,\,\,\,\,\,\,\,\,\mspace{6mu}( - 2 \times \log - likelihood).$

The quantity -2 × the log-likelihood may be referred to as the deviance.

Again, test error here may be ErrT = E[L(G, G^(X)))|T], the populationmis- classification error of the classifier trained on T , and Err maybe the expected misclassification error of not predicting hypoglycemia.

Training error may be the sample analogue, for example,

$err = - (2/N){\sum{Ni = 1\,\,\,\,\,\,\,\,\log\,\, p^{\hat{}}gi(xi),}}$

which is the sample log-likelihood for the model.

In embodiments, models may have a tuning parameter or parameters α wherethe prediction may be written as f^a (x). The tuning parameter may varythe complexity of a model and it is imperative to minimize error. Thismay include two aspects including estimating the performance ofdifferent models in order to choose the best one and estimating amodel’s prediction error (generalization error) on new data.

While method 200 is described as splitting the total data points intotwo sets to use as learning data and testing data, in some embodiments,the data points may be divided into three sets: a training set, avalidation set, and a test set. The training set may be used to fit themodels, the validation set may be used to estimate prediction error formodel selection, and the test set may be used for assessment of thegeneralization error of the final chosen model. Ideally, the test setmay be only used for data analysis. That is, if a first test-set is usedrepeatedly with various models, then the model with the model withsmallest test-set errors may be chosen. Then the test set error of thefinal chosen model may underestimate the true test error, sometimessubstantially because of its reliance on only the first test-set. Insome additional embodiments, the data points may be divided in four ormore sets as well.

In embodiments, an assessment of generalization performance of alearning method is very importance since it guides the choice oflearning method or model that is selected. Additionally, the assessmentprovides a measure of the quality of the chosen model.

While FIG. 2 describes an exemplary method for creating one or moremodels. Additional methods may be utilized for creating the library ofmodels consistent with exemplary embodiments. Furthermore, exemplarymethod 300 described below does not require that the library of modelsbe necessarily created using exemplary method 200.

FIG. 3 is a flow diagram of a method 300 for selecting one or moremodels predicting medical conditions including adverse health events,according to an embodiment of the present disclosure.

In step 302, method 300 includes receiving data related to a patient.For example, this data may be received and/or retrieved by processor 110from one or more databases 108 or may be directly received from one ormore user or client devices 102. The data may include clinical,behavioral, personal and system data. For example, self-reported data,data from one or more sensors (such as blood glucose testing machines),patient data from historical electronic records, secondary sources ofdata, system logs, and system-reported data that is based on inferenceor machine learning, etc.

In embodiments, self-reported data may include data that is provided bya user, for example, blood glucose levels and associated time stampsself-reported by the patient (SMBG) using one of client or user devices102.

In embodiments, one or more sensors may include sensors related topatient’s medical conditions or other related sensors. For example,blood glucose levels may be monitored from continuous glucose monitoring(CGM) devices for hypoglycemic patients. Similarly, pulmonary patientsmay have oxygen level monitors, etc. Additional sensors may provideinformation such as location information from mobile devices, weatherinformation from publicly available data sources, elevation information,etc.

In embodiments, secondary sources of data may include informationrelated to a patient that may be provided by other individuals orentities such as caregivers. This data may include subjectiveobservations regarding patient behavior and moods. Additionally, systemlogs such as data from mobile applications may be utilized. The systemslogs may provide information including when a first health monitoringapplication started being used by a patient, how it has been used over acertain time period, etc. As another example, an exercise monitoringapplication may provide data related to a patient’s exercise.

System-reported data that is based on inference or machine learning mayinclude utilizing data related to certain conditions and determiningthat other conditions may exist based on machine learning. For example,low levels of exercise and certain vital signs may indicate thepossibility of depression or a probability of a depressive episode.

This received data may previously be generated utilizing processor 110and stored in databases 108. Alternatively, the received data may betransmitted by one of user or client devices 102, including datapreviously transmitted and stored in databases 108. In embodiments, oneof user or client devise 102 may transmit data in real time,continuously, periodically, or based on a predetermined duty cycle.

In step 304, method 300 includes extracting metadata from the receiveddata. In detail, metadata may be extracted from the received data toextract information about data such as descriptive statistics,availability, data type, category type, degree of relationship and/orinfluence.

In embodiments, descriptive statistics may include mean, standarddeviation, variance, etc. of the objective data that is received.Objective data may include numerical values related to vital signs,nutrition consumed, and medication consumed, etc.

Availability may refer to determining when different types of data isavailable. For example, certain days SMBG data may have been input by ahypoglycemic patient and therefore may be available. For thoseinstances, frequency information may also be extracted. Frequencyinformation allows for picking of models which are more efficient forthe amount of data that is available. For other days, CGM data may beavailable and that data is utilized. As an additional example ofdifferent types of data, the different types of data may below-frequency data (such as data from a lab setting) as opposed tohigh-frequency data (such as captured by home monitoring devices.)

Data type may refer to the types of data that is received. For example,the received data may related to conditions external to the patient suchas location and weather. Accordingly, this extracted metadata may allowfor selection of appropriate location specific models orweather-specific models. The selection process is explained in furtherdetail below with respect to the explanation of step 306.

Category type may refer to whether the received data is clinical,behavioral, or personal. Clinical data may include data that is capturedin a clinical setting including a visit to a doctor’s office or resultsfrom a medical lab. Behavioral data may include data related to apatient’s behavior such as sleeping, exercising, eating, patient’s mood,and consuming medications, etc. Personal data may include, but is notlimited to, data related to the patient including weight, height, bodyfat, vital signs, and medical history, etc.

Degree of relationships and/or influence on other categories may referto contextualization of received data. In an embodiment, if a certaincategory is selected, then there may be an assigned value to a relatedcategory. For example, data related to a person’s weight may be receivedbut if such an increase correlates with an increase in a blood pressurelevel, a value may be assigned to the blood pressure level even thoughno data is received directly providing a blood pressure level.

In embodiments, metadata may be extracted based on categories of storedlibrary of models. Alternatively, the received data may be processed forextraction using batch-based data or streaming data. For example, rulesengines or a XML parser may be used for extracting the metadata from thereceived data.

In step 306, method 300 may include selecting one or more models from alibrary of models based on the extracted metadata. For example,processor 110 may apply rules using the extracted metadata to determinethe one or more models to select from a library of models that aregenerated by method 200 described above. The rules may be simplelogical, mathematical, algorithmic rules, or machine learned models.These rules may be applied to models that are stored in databases 108.In embodiments, a library of models may be referred to as logicallibrary (LL). Models may be stored in the logical library (LL)independently and/or may be categorized per one or more criteria. Forexample, the categories may be clinical, behavioral, or personal data.However, other suitable categorization may be applied.

In additional embodiments, a plurality of models or a combination ofmodels are to be selected. In instances when more than one combinationexists for a given context, potential conflicts may be resolved byfollowing certain rules of selection. For example, rules of selectionmay provide limitations for picking rules and/or models. The rules mayinclude limitations regarding allowable and non-allowable combinationsof models. The rules also may prioritize a list of eligible models orrules that may be applied. In embodiments, a threshold number of highlyprioritized models may be selected. The rules of selection also may bebased on randomized logic. The rules may be specific or agnostic of agiven user or population.

In step 308, method 300 may include applying the selected one or models.For example, the selected models may be applied to received data or theextracted metadata to predict medical occurrences. An exemplary use caseis described below with respect to FIG. 4 providing further insight intothis step.

In step 310, method 200 may include generating a notification when theapplication of the selected model indicates an intervention may beneeded. For example, for a hypoglycemia patient, when it appears that apatient’s blood glucose level will likely drop based on application ofthe model, a notification may be generated. The generated notificationmay be provided to the patient or other concerned entities such aspharmacists, physicians, and clinicians, etc. The notifications mayserve as reminders as well as instruments to force interventions. Forexample, a prediction of a critical medical condition may trigger anemergency call from the physician’s office to the patient. If thepatient is not responsive, the medical service provider may contact apatient’s emergency contact person and/or local emergency medicalservices entities.

In some embodiments, a motivation may trigger a clinical or behavioralaction such as medication dosage change or motivational messaging.Additionally, a patient may receive information regarding correctingactions that may be taken to reduce the probability of experiencing anadverse health event. Therefore, a patient may take a corrective actionsuch as changing their diet, resting, or consuming a medication to aidin preventing or reducing the probability of an adverse health event.

In some embodiments, the metadata does not need to be extracted. In sucha scenario, rules may be stored in databases 108 that allow for one ormore models to be selected from the library of models based on thereceived data.

FIG. 4 presents an exemplary use case applying the principles of method300 presented in FIG. 3 above, according to an embodiment of the presentdisclosure. One of ordinary skill in the art would comprehend that theboxes 402-410 illustrated in FIG. 4 do not represent structurallimitations. Rather, boxes 402-410 represent possible implementations ofcombinations of one or more of the steps of method 300 that are executedby processor 110.

This use cases deals with a prediction of whether a patient may slip into a coma (a prolonged state of unconsciousness) within a given calendarweek. A data storage/stream service (SS) 402 may store and/or receivedata related to the patient. For example, the stored data may includeinformation related to a patient including the fact that the patient hasType 2 diabetes and has been diagnosed with severe depression. Thisinformation may have been previously received and stored withindatabases 108 as being associated with the patient. Additional receiveddata by SS 402 may include self-reported data such as blood glucoselevels and associated time stamps self-reported by the patient (SMBG).Additionally, information from one or more sensors may be received suchas blood glucose levels from a continuous glucose monitoring (CGM)device. Furthermore, information received from one or more sensors mayinclude location information from mobile devices or weather informationfrom publicly available data sources. Additionally, information mayinclude secondary sources of data, such as information provided bycaregivers. For example, with regards to depression for this exemplaryuse case, the patient may provide the primary data regarding theirsymptoms while the caregiver may provide secondary data includingobservations regarding the patient.

The received data may be provided to metadata extractor service (MES)404 for extracting metadata from the data received by SS 402. MES 404may extract metadata from the received data. For example, for days whenSMBG is available, MES 404 may extract frequency information andspecifies that the received data is sparsely distributed. However, fordays when CGM data is available, MES 404 may specify that the receiveddata is densely distributed.

MES 404 may then send the extracted metadata information to logicprocessing service (LPS) 406. LPS 406 may use the metadata informationto decide which models to pick based on pre-defined rules. For example,LPS 406 may choose to pick SMBG-specific model for cases when SMBG datais available. Additional examples of rules regarding picking models mayinclude LPS 406 taking numerous additional factors into considerationfor picking the appropriate model(s) including data reported by patientversus those reported by a caregiver, location-specific models based onlocation information from mobile device, and weather-specific modelsbased on received weather information.

While in many instances, output of LPS 406 may be sufficient to selectthe models, it may be important to check if there are any conflictsbetween selecting multiple models and if there are combinations ofmodels that may be chosen. In this example, since the patient has twodiseases of diabetes and depression, the chosen models must beapplicable for both without any major issues. It is possible that thesymptoms reported by patient may conflict with the information reportedby the caregiver. Accordingly, in this exemplary use case, modelcombinator service (MCS) 408 may be utilized to solve such a possibleconflict by providing predefined rules to prioritize patient provideddata over date supplied by, e.g., a caregiver, or vice-versa dependingon context.

In embodiments, to select the correct models based on the requirementsfrom MCS 408, LL 410 may be queried and models applicable for bothdiseases and models that take weather, location, and symptoms, or anyother suitable factors, into account, may be retrieved and provided asoutput 412.

The output 412 may be utilized to generate notifications when a medicalintervention may be necessary. In this instance, if it appears that thepatient may go into coma, a notification may be generated. Thisnotification may be provided to the patient, caregiver, physician,physician’s assistant, etc. For example, a diabetes patient may get anotification that unless they inject insulin immediately, they are atrisk for an hypoglycemic episode.

In embodiments, exemplary systems consistent with the present disclosuremay be designed to perform automatic self-maintenance and tuning withregards to the various models. For example, the exemplary systems may beable to identify errors and modify rules associated with selection ofmodels to minimize error over time. Additionally, new possible modelsmay be automatically evaluated and considered to be added to the pool ofmodels for selection. Furthermore, machine learning algorithms may beused for learning about new types of data that may be received andrecommend possible model selection logic for its predictions, based onperformance of such models on similar data.

In embodiments, stochastic models may be utilized. However, theirresults may be transformed into probabilistic and/or deterministicterms. For example, all of the data first may be transformed intodeterministic data. The transformation may occur based on inputs by adata analyst, a particular set of rules defining the transformation, ora machine learned algorithm. Once, all the data has been transformedinto deterministic data, all the data may be combined since it is in thesame format. Using deterministic data allows experts trained in thefield to specify the rules for selection of models in an easier manner.

The examples described above with respect to FIGS. 1-4 , or any part(s)or function(s) thereof, may be implemented using hardware, softwaremodules, firmware, tangible computer readable media having instructionsstored thereon, or a combination thereof and may be implemented in oneor more computer systems or other processing systems.

In embodiments, being able to predict an occurrence of an adversemedical event, such as a hypoglycemic event may have many benefits. Forexample, a pay/self-insured employer may be interested in screeningpatients who have high risk of hypoglycemia within a forecasted periodin order to adjust premium cost of health insurance or provide foradditional care. Alternatively, a hospital may be interested inidentifying high-risk patient to prevent instances of readmission.Similarly, an insurance payer may use predictions to aid in preventingexpensive emergency room visits. Additionally, guidance, including inthe form of messages and suggestions, may be provided to a patient basedon the predictions including suggestions to diet, medication regimen, ornutrition, etc.

FIG. 5 illustrates a high-level functional block diagram of an exemplarycomputer system 500, in which embodiments of the present disclosure, orportions thereof, may be implemented, e.g., as computer-readable code.For example, each of the exemplary devices and systems described abovewith respect to FIG. 1 can be implemented in computer system 500 usinghardware, software, firmware, tangible computer readable media havinginstructions stored thereon, or a combination thereof and may beimplemented in one or more computer systems or other processing systems.Hardware, software, or any combination of such may embody any of themodules and components in FIG. 1 , as described above.

If programmable logic is used, such logic may be executed on acommercially available processing platform or a special purpose device.One of ordinary skill in the art may appreciate that embodiments of thedisclosed subject matter can be practiced with various computer systemconfigurations, including multi-core multiprocessor systems,minicomputers, mainframe computers, computers linked or clustered withdistributed functions, as well as pervasive or miniature computers thatmay be embedded into virtually any device.

For instance, at least one processor device and a memory may be used toimplement the above-described embodiments. A processor device may be asingle processor, a plurality of processors, or combinations thereof.Processor devices may have one or more processor “cores.”

Various embodiments of the present disclosure, as described above in theexamples of FIGS. 1-4 , may be implemented using computer system 500.After reading this description, it will become apparent to a personskilled in the relevant art how to implement embodiments of the presentdisclosure using other computer systems and/or computer architectures.Although operations may be described as a sequential process, some ofthe operations may in fact be performed in parallel, concurrently,and/or in a distributed environment, and with program code storedlocally or remotely for access by single or multi-processor machines. Inaddition, in some embodiments the order of operations may be rearrangedwithout departing from the spirit of the disclosed subject matter.

As shown in FIG. 5 , computer system 500 includes a central processingunit (CPU) 520. CPU 520 may be any type of processor device including,for example, any type of special purpose or a general-purposemicroprocessor device. As will be appreciated by persons skilled in therelevant art, CPU 520 also may be a single processor in amulti-core/multiprocessor system, such system operating alone, or in acluster of computing devices operating in a cluster or server farm. CPU520 may be connected to a data communication infrastructure 610, forexample, a bus, message queue, network, or multi-core message-passingscheme.

Computer system 500 also may include a main memory 540, for example,random access memory (RAM), and also may include a secondary memory 530.Secondary memory 530, e.g., a read-only memory (ROM), may be, forexample, a hard disk drive or a removable storage drive. Such aremovable storage drive may comprise, for example, a floppy disk drive,a magnetic tape drive, an optical disk drive, a flash memory, or thelike. The removable storage drive in this example reads from and/orwrites to a removable storage unit in a well-known manner. The removablestorage unit may comprise a floppy disk, magnetic tape, optical disk,etc. which is read by and written to by the removable storage drive. Aswill be appreciated by persons skilled in the relevant art, such aremovable storage unit generally includes a computer usable storagemedium having stored therein computer software and/or data.

In alternative implementations, secondary memory 530 may include othersimilar means for allowing computer programs or other instructions to beloaded into computer system 500. Examples of such means may include aprogram cartridge and cartridge interface (such as that found in videogame devices), a removable memory chip (such as an EPROM, or PROM) andassociated socket, and other removable storage units and interfaces,which allow software and data to be transferred from a removable storageunit to computer system 500.

Computer system 500 also may include a communications interface (“COM”)560. Communications interface 560 allows software and data to betransferred between computer system 500 and external devices.Communications interface 660 may include a modem, a network interface(such as an Ethernet card), a communications port, a PCMCIA slot andcard, or the like. Software and data transferred via communicationsinterface 560 may be in the form of signals, which may be electronic,electromagnetic, optical, or other signals capable of being received bycommunications interface 560. These signals may be provided tocommunications interface 560 via a communications path of computersystem 600, which may be implemented using, for example, wire or cable,fiber optics, a phone line, a cellular phone link, an RF link or othercommunications channels.

The hardware elements, operating systems, and programming languages ofsuch equipment are conventional in nature, and it is presumed that thoseskilled in the art are adequately familiar therewith. Computer system500 also may include input and output ports 550 to connect with inputand output devices such as keyboards, mice, touchscreens, monitors,displays, etc. Of course, the various server functions may beimplemented in a distributed fashion on a number of similar platforms,to distribute the processing load. Alternatively, the servers may beimplemented by appropriate programming of one computer hardwareplatform.

Program aspects of the technology may be thought of as “products” or“articles of manufacture” typically in the form of executable codeand/or associated data that is carried on or embodied in a type ofmachine-readable medium. “Storage” type media include any or all of thetangible memory of the computers, processors or the like, or associatedmodules thereof, such as various semiconductor memories, tape drives,disk drives and the like, which may provide non-transitory storage atany time for the software programming. All or portions of the softwaremay at times be communicated through the Internet or various othertelecommunication networks. Such communications, for example, may enableloading of the software from one computer or processor into another, forexample, from a management server or host computer of the mobilecommunication network into the computer platform of a server and/or froma server to the mobile device. Thus, another type of media that may bearthe software elements includes optical, electrical and electromagneticwaves, such as used across physical interfaces between local devices,through wired and optical landline networks and over various air-links.The physical elements that carry such waves, such as wired or wirelesslinks, optical links, or the like, also may be considered as mediabearing the software. As used herein, unless restricted tonon-transitory, tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

It would also be apparent to one of skill in the relevant art that thepresent disclosure, as described herein, can be implemented in manydifferent embodiments of software, hardware, firmware, and/or theentities illustrated in the figures. Any actual software code with thespecialized control of hardware to implement embodiments is not limitingof the detailed description. Thus, the operational behavior ofembodiments will be described with the understanding that modificationsand variations of the embodiments are possible, given the level ofdetail presented herein.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the disclosed embodiments, as claimed.

Other embodiments of the disclosure will be apparent to those skilled inthe art from consideration of the specification and practice of theinvention disclosed herein. It is intended that the specification andexamples be considered as exemplary only, with a true scope and spiritof the invention being indicated by the following claims.

1-20. (canceled)
 21. A computer-implemented method for creating a library of models for predicting medical conditions, the method comprising: receiving, using a processor, initial data related to a user and other data related to a medical condition of the user; processing, using the processor and one or more machine learning algorithms, at least a portion of the initial data and at least a portion of the other data related to the medical condition of the user to identify patterns between the initial data and the other data related to the medical condition of the user and to generate at least one model; applying at least another portion of the initial data and at least another portion of the other data related to the medical condition of the user to test the at least one generated model; evaluating the at least one generated model based on the applying of at least the other portion of the initial data and at least the other portion of the other data related to the medical condition of the user; and storing the at least one generated model to thereby create the library of models.
 22. The computer-implemented method of claim 21, wherein the initial data includes at least one of data based on self-reporting by the user or data from a user device that automatically obtains and transmits the data.
 23. The computer-implemented method of claim 21, further comprising cleansing the received initial data and the other data related to the medical condition of the user to generate cleansed initial data.
 24. The computer-implemented method of claim 23, wherein the cleansing includes fixing one or more of anomalies, errors, missing values, or large number ranges determined to be in the received initial data and the other data related to the medical condition of the user.
 25. The computer-implemented method of claim 24, wherein the fixing the one or more of anomalies, errors, missing values, or large number ranges includes fixing missing values by replacing the missing values with zero.
 26. The computer-implemented method of claim 23, wherein the received initial data includes at least blood glucose levels of the user, associated time stamps, and other data, and wherein the cleansing includes removing the other data.
 27. The computer-implemented method of claim 23, wherein the processing at least the portion of the initial data and at least the portion of the other data related to the medical condition of the user includes processing the cleansed initial data.
 28. The computer-implemented method of claim 27, wherein the cleansed initial data includes between 10% and 50% of the initial data and the other data related to the medical condition of the user.
 29. The computer-implemented model of claim 21, wherein the applying at least the other portion of the initial data and at least the other portion of the other data related to the medical condition of the user to test the at least one generated model includes applying the one or more machine-learning algorithms to predict a prediction variable with a level of confidence, the level of confidence indicating an accuracy of the prediction variable.
 30. The computer-implemented method of claim 29, wherein the one or more machine learning algorithms include one or more of support vector machines, k-nearest neighbor, Bayesian statistics, multi-layer perceptrons, and multivariate regressions.
 31. The computer-implemented method of claim 29, wherein the evaluating the at least one generated model includes generating a confusion matrix.
 32. A system for creating a library of models for predicting medical conditions, the system comprising: a memory having processor-readable instructions stored therein; and a processor configured to access the memory and execute the processor-readable instructions, which, when executed by the processor configures the processor to perform a method, the method comprising: receiving, using a processor, initial data related to a user and other data related to a medical condition of the user; processing, using the processor and one or more machine learning algorithms, at least a portion of the initial data and at least a portion of the other data related to the medical condition of the user to identify patterns between the initial data and the other data related to the medical condition of the user and to generate at least one model; applying at least another portion of the initial data and at least another portion of the other data related to the medical condition of the user to test the at least one generated model; evaluating the at least one generated model based on the applying of at least the other portion of the initial data and at least the other portion of the other data related to the medical condition of the user; and storing the at least one generated model to thereby create the library of models.
 33. The system of claim 32, wherein the initial data includes at least one of data based on self-reporting by the user or data from a user device that automatically obtains and transmits the data.
 34. The system of claim 32, further comprising cleansing the received initial data and the other data related to the medical condition of the user to generate cleansed initial data.
 35. The system of claim 34, wherein the cleansing includes fixing one or more of anomalies, errors, missing values, or large number ranges determined to be in the received initial data and the other data related to the medical condition of the user.
 36. The system of claim 35, wherein the fixing the one or more of anomalies, errors, missing values, or large number ranges includes fixing missing values by replacing the missing values with zero.
 37. The system of claim 34, wherein the received initial data includes at least blood glucose levels of the user, associated time stamps, and other data, and wherein the cleansing includes removing the other data.
 38. The computer-implemented method of claim 34, wherein the processing at least the portion of the initial data and at least the portion of the other data related to the medical condition of the user includes processing the cleansed initial data.
 39. The computer-implemented method of claim 38, wherein the cleansed initial data includes between 10% and 50% of the initial data and the other data related to the medical condition of the user.
 40. A non-transitory computer-readable medium storing instructions, the instructions, when executed by a computer system cause the computer system to perform a method for creating a library of models for predicting medical conditions, the method comprising: receiving, using a processor, initial data related to a user and other data related to a medical condition of the user; processing, using the processor and one or more machine learning algorithms, at least a portion of the initial data and at least a portion of the other data related to the medical condition of the user to identify patterns between the initial data and the other data related to the medical condition of the user and to generate at least one model; applying at least another portion of the initial data and at least another portion of the other data related to the medical condition of the user to test the at least one generated model; evaluating the at least one generated model based on the applying of at least the other portion of the initial data and at least the other portion of the other data related to the medical condition of the user; and storing the at least one generated model to thereby create the library of models. 